Analytical engineering and AI: How ChatGPT is transforming the data industry from Coalesce 2023

"AI is going to happen. It's going to take on more responsibilities. It's going to keep improving."

- Kate Schiffelbein, Head of Business Intelligence at Northbeam

Kate Schiffelbein, Head of Business Intelligence at Northbeam, Lindsay Murphy, Head of Data at Secoda, and Patrick Ross, Solutions Architect at Clymer, discuss the impact of AI on the role of analytics engineers (AEs). The panelists cover the hype around AI replacing jobs, the reality of AI's current capabilities, and the potential future of AI in the field of analytics. They also consider the risks and opportunities associated with implementing AI tools like ChatGPT in data workflows.

AI is creating new opportunities but isn't replacing jobs entirely

Contrary to the initial hype, AI hasn't replaced all our jobs. Instead, it’s merely changed the nature of our jobs and created new opportunities. Lindsay Murphy and Pat Ross both concurred that while AI has its limitations, it can make certain tasks easier and free analytics engineers to focus on more complex issues.

"AI is still a thing. Data engineering is still a thing,” says Kate. Lindsay adds, “There are a lot of aspects to it that aren't so glamorous... I think there's a lot of places where ChatGPT can really help with certain things...making some of those more manual, tedious tasks a little bit easier.” Patrick also notes, "I mean, who among us hasn't tried asking ChatGPT to write an API call for you or something tedious like that? It's not really at that point yet where it's going to replace your ability to write your own code."

Lindsay believes that the implementation of AI tools might change the definition of “analytics engineer” and open up more opportunities. "...thinking about what the definition of the role actually is and how that's going to change over time... opens up more opportunities for us," she says.

The implementation of AI tools presents risks that need to be managed

While the possibilities presented by AI are exciting, the implementation of these tools also carries certain risks, particularly in terms of data quality and decision-making.

“You’ve got to make sure you really understand what platform you're using and what data is being fed into that platform in order to make those decisions," says Kate. She also warns against blindly trusting AI to make business decisions, especially if the data quality isn't up to par.

Lindsay echoes this sentiment, noting that the rapid implementation of AI could actually stall the progress of companies trying to become more data-driven. She stresses the need to treat AI as a tool that requires careful implementation and management.

AI is changing the day-to-day workflow of data teams and creating new roles

"In its current stage, it's really more of an augmentation tool than a replacement for an analytics engineer."

• Patrick Ross, Solutions Architect at Data Clymer

AI is not just a tool that augments existing roles; it's also creating new ones. Lindsay, Kate, and Patrick discuss the emergence of roles like prompt engineering and the need for new skills. Lindsay suggests that AI might change how data is modeled, while Patrick sees an increased demand for the profession.

"I think there's going to be a need for new skill sets, including prompt engineering," says Patrick. Lindsay adds, "I sort of think of it as…almost like a junior engineer... You do really want to have that scrutiny and double-check the work that it's producing."

Kate also emphasizes the role of managers in ensuring that their team members grow and adapt to these changes. She says, "I consider that a very important part of being a manager... How am I making sure that they're best positioned, knowing that this is coming down the pipeline?"

Insights surfaced

• AI tools like ChatGPT have not replaced jobs as hyped, but they have affected the tools and features used in different ways
• AI is seen more as an augmentation tool rather than a replacement tool for AEs
• There are risks associated with implementing AI tools, such as data privacy and quality issues
• AI tools can help automate some of the more tedious tasks in data engineering, allowing AEs to focus on more complex tasks
• AI implementation can change the role of data engineers and create new roles and opportunities in the field
• The use of AI tools requires new skills and a shift in the day-to-day workflow of analytics engineers